Image pickup apparatus, an image processing method and a non-transitory computer-readable medium

ABSTRACT

Image pickup apparatuses, control methods and storage mediums for use therewith are provided herein. At least one image pickup apparatus includes: a display unit that displays an image generated by an imaging unit; an image processing unit that extracts in-focus areas from a plurality of images different in in-focus position that are generated by the imaging unit and that composites the extracted areas; and a control unit. The control unit acquires, out of image magnifications corresponding to the plurality of images different in in-focus position, a reference image magnification, and the image processing unit corrects the image to be displayed on the display unit based on an image magnification corresponding to the image to be displayed on the display unit and the reference image magnification.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation, and claims the benefit, of U.S.patent application Ser. No. 16/594,358, presently pending and filed onOct. 7, 2019, which is a continuation, and claims the benefit, of U.S.patent application Ser. No. 15/838,135, filed on Dec. 11, 2017 andissued as U.S. Pat. No. 10,484,612 on Nov. 19, 2019, and claims thebenefit of, and priority to, Japanese Patent Application No.2016-248205, filed Dec. 21, 2016, which applications are herebyincorporated by reference herein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to one or more embodiments of a displayon an image pickup apparatus, in particular to the display of imagescaptured at different in-focus positions.

Description of the Related Art

In the case of capturing images of a plurality of subjects at differentdistances from an image pickup apparatus such as a digital camera, or inthe case of capturing an image of a subject that is long in a depthdirection, only part of the subject may come into focus due toinsufficient depth of field in an imaging optical system. To solve thisproblem, Japanese Patent Laid-Open No. 2015-216532 discloses a techniquefor generating a composite image being in-focus in the entire imagingarea by capturing a plurality of images at different in-focus positions,extracting in-focus areas from the images, and compositing the in-focusareas into one image.

However, when the foregoing image capturing is performed using live-viewdisplay of an image pickup apparatus such as a digital camera, fieldangles of the captured images may change with alterations of in-focusposition due to the changing lens feed amount or the like. To generate acomposite image, with reference to the field angle of the captured imagewith the narrowest field angle, the other captured images need cut-outprocessing. Accordingly, the field angle of the live view image beingcaptured may be different from the field angle of the composite image.

Japanese Patent Laid-Open No. 2015-231058 discloses a method forgenerating image data for verification through simple pre-imaging andimage processing, prior to real imaging.

However, according to the method described in Japanese Patent Laid-OpenNo. 2015-231058, it is necessary to perform the pre-imaging prior to thereal imaging, which causes trouble to the person in charge of imagingand lengthens the entire time of imaging operation.

SUMMARY OF THE INVENTION

At least one object of the present disclosure is to provide at least oneembodiment of an image pickup apparatus that allows capturing of aplurality of images different in in-focus position and verification ofthe field angle of a composite image with convenience.

The present disclosure provides at least one embodiment of an imagepickup apparatus including: an imaging unit; a display unit thatdisplays an image generated by the imaging unit; an image processingunit that extracts in-focus areas from a plurality of images differentin in-focus position that are generated by the imaging unit and thatcomposites the extracted areas; and a control unit. The control unitcorrects the image to be displayed on the display unit based on, out ofimage magnifications corresponding to the plurality of images differentin in-focus position, a reference image magnification.

According to other aspects of the present disclosure, one or moreadditional image pickup apparatuses, one or more image pickup methodsand one or more storage mediums for use therewith are discussed herein.Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of structure of a digital camera according toembodiments of the present disclosure.

FIG. 2 is a flowchart for describing at least a first embodiment.

FIG. 3 is a diagram for describing a function indicating a relationshipbetween an in-focus position and an image magnification in at least thefirst embodiment.

FIGS. 4A and 4B are diagrams for describing a maximum setting imagemagnification and an LV image magnification in at least the firstembodiment.

FIGS. 5A and 5B are diagrams for describing examples of an array ofsensors included in imaging elements capable of acquiring rangeinformation on a subject.

FIG. 6 is a diagram for describing incidence of optical signals on apixel having a plurality of photoelectric conversion units.

FIGS. 7A, 7B, and 7C are diagrams for describing a method for acquiringrange information on a subject by using pixels having a plurality ofphotoelectric conversion units.

FIG. 8 is a flowchart for describing at least a second embodiment.

FIG. 9A is a diagram illustrating a monotonic decrease of an imagemagnification, and FIG. 9B is a diagram illustrating a monotonicincrease of an image magnification.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present disclosure will be described indetail with reference to the attached drawings.

First Embodiment

FIG. 1 is a block diagram of structure of a digital camera according toa first embodiment of the present disclosure.

A control circuit 101 is, for example, a signal processor such as a CPUor an MPU, that reads in advance programs from a ROM 105 described laterto control the respective components of a digital camera 100. Forexample, as described later, the control circuit 101 provides commandsfor starting and terminating of image capturing to an imaging element104 described later. In addition, the control circuit 101 providescommands for image processing to an image processing circuit 107described later based on the programs stored in the ROM 105. Thecommands from the user are input into the digital camera 100 by anoperation member 110 described later and transmitted to the respectivecomponents of the digital camera 100 via the control circuit 101.

A drive mechanism 102 is formed from a motor or the like and causesmechanical operation of an optical system 103 described later, under thecommands from the control circuit 101. For example, under a command fromthe control circuit 101, the drive mechanism 102 moves the position of afocus lens included in the optical system 103 to adjust the focal lengthof the optical system 103.

The optical system 103 is formed from a zoom lens, a focus lens, adiaphragm, and the like. The diaphragm is a mechanism for adjusting theamount of transmitted light. The in-focus position can be changed bychanging the position of the focus lens.

The imaging element 104 is a photoelectric conversion element thatperforms photoelectric conversion of an incident optical signal into anelectric signal. For example, the imaging element 104 may be a CCD orCMOS sensor, or the like.

The ROM 105 is a read-only volatile memory as a recording medium thatstores not only operation programs for the blocks included in thedigital camera 100 but also parameters and the like necessary foroperation of the blocks. The RAM 106 is a rewritable volatile memorythat is used as a temporary storage area for data output during theoperation of the blocks included in the digital camera 100.

The image processing circuit 107 performs various kinds of imageprocessing, such as white balance adjustment, color interpolation, andfiltering, on the image output from the imaging element 104 or the dataof the image signal recorded in an internal memory 109 described later.In addition, the image processing circuit 107 performs compressionprocessing on the data of the image signal captured by the imagingelement 104 under standards such as JPEG.

The image processing circuit 107 is formed from an integrated circuit(ASIC) in which circuits performing specific processes are united.Alternatively, the control circuit 101 may perform processing accordingto the programs read from the ROM 105 so that the control circuit 101also performs some or all of the functions of the image processingcircuit 107. When the control circuit 101 also performs all thefunctions of the image processing circuit 107, it is not necessary tohave the image processing circuit 107 as hardware.

A display 108 is a liquid crystal display or an organic EL display thatdisplays images temporarily saved in the RAM 106, or images saved in theinternal memory 109 described later, or setting screens of the digitalcamera 100. The display 108 can reflect the image acquired by theimaging element 104 as a display image in real time, and present theimage in live view display.

The internal memory 109 is a place that records the images captured bythe imaging element 104, the images processed by the image processingcircuit 107, and information on the in-focus positions at the time ofimage capturing. Instead of the internal memory, a memory card or thelike may be used.

The operation member 110 includes buttons, switches, keys, mode dial inthe digital camera 100, or a touch panel also used as the display 108,for example. The commands input by the user using the operation member110 reach the control circuit 101, and the control circuit 101 controlsthe operations of the blocks in response to the commands.

FIG. 2 is a flowchart for describing the first embodiment.

At step S201, the control circuit 101 acquires lens information. Forexample, in a digital camera incapable of lens replacement, the lensinformation is saved in advance in the ROM 105, and the control circuit101 acquires the lens information at S201. Alternatively, in a digitalcamera capable of lens replacement, the control circuit 101 reads thetype of the lens through a mount provided on the optical system 103 orthe like. The information by lens type is saved in advance in the ROM105, and the control circuit 101 compares the information with the readlens type, and reads the relevant lens information from the ROM 105. Thelens information here refers to unique information on the lens, such asa relationship between an in-focus position and an image magnificationin the case of using a lens described later.

In the first embodiment, the control circuit 101 reads a functionindicating the relationship between an in-focus position and an imagemagnification included in the lens information. The image magnificationand the field angle of an image are in inverse proportion to each other.Accordingly, the control circuit 101 can also acquire the relationshipbetween an in-focus position and a field angle from the functionindicating the relationship between the in-focus position and the imagemagnification. In general, the function indicating the relationshipbetween the in-focus position and the image magnification is uniquelydetermined by the type of the lens, having a monotonic change. The imagemagnification here refers to the ratio of the size of a subject imageformed by the imaging element to the size of the actual subject.

FIG. 3 is a diagram for describing a function indicating therelationship between the in-focus position and the image magnificationin the first embodiment. The function described in FIG. 3 has amonotonic decrease with respect to the image magnification.

At step S202, the control circuit 101 sets a plurality of in-focuspositions. For example, the user sets a focus area using the touchpanel, and the optical system 103 measures an in-focus positioncorresponding to the focus area. The control circuit 101 sets apredetermined number of in-focus positions in front of and behind themeasured in-focus position. The control circuit 101 preferably sets thedistance between the adjacent in-focus positions such that the endportions of their field depths slightly overlap.

At step S203, the control circuit 101 calculates the imagemagnifications. At step 203, the control circuit 101 uses the functionindicating the relationship between the in-focus position and the imagemagnification acquired at step S201 to calculate the imagemagnifications corresponding to the in-focus positions set at step S202.For example, as illustrated in FIG. 3, the control circuit 101 setsin-focus positions 301 to 304 at step S202. According to the functionillustrated in FIG. 3, it can be understood that, when the opticalsystem 103 obtains focus at the in-focus positions 301 to 304, the imagemagnifications of the images captured by the imaging element 104 arerespectively image magnifications 311 to 314.

At step S204, the control circuit 101 compares a maximum setting imagemagnification to a live view image magnification (hereinafter, called LVimage magnification). The maximum setting image magnification hererefers to the highest image magnification calculated at step S203, whichare included in a reference image magnification. The LV imagemagnification refers to the image magnification corresponding to thein-focus position set for live view display on the display 108.

When determining at step S204 that the maximum setting imagemagnification is higher than the LV image magnification, the controlcircuit 101 moves onto step S205, and when not determining so, thecontrol circuit 101 moves onto step S206.

FIGS. 4A and 4B are diagrams for describing the maximum setting imagemagnification and the LV image magnification in the first embodiment.FIG. 4A illustrates the field angle of an image with the LV imagemagnification, and FIG. 4B illustrates the field angle of an image withthe maximum setting image magnification. The field angle of an image 401illustrated in FIG. 4A is wider than the field angle of an image 402illustrated in FIG. 4B, and the field angle of a portion in a frame 403illustrated in FIG. 4A is equivalent to the field angle of the image402. Therefore, in an attempt of compositing the image 401 and the image402, the control circuit 101 needs to abandon the portion of the image401 outside the frame 403, and the generated composite image would haveno portion outside the frame 403. The display 108 has the image 401 withthe portion outside the frame 403 on the live view display, whereas thedisplay 108 displays the composite image without that portion. To solvethis problem, when the control circuit 101 determines that the maximumsetting image magnification is higher than the LV image magnification,the live view image is corrected so that the display 108 does not havethe portion of the image 401 outside the frame 403 on live view display.Specifically, at step S205, the image processing circuit 107 cuts outthe portion within the frame 403 of the image 401 illustrated in FIG. 4Aand enlarges the cut-out portion to the original size of the image 401.

At step S206, the display 108 displays the live view image. When theimage processing circuit 107 made live view correction at step S205, atstep S206 the display 108 has the post-correction image as display imageon live view display, not the pre-correction image.

At step S207, the control circuit 101 determines whether there is aninstruction for starting imaging from, for example, the user operating abutton on the operation member 110. When there is the instruction forstarting imaging, the control circuit 101 moves onto step S207. Whenthere is no instruction for starting imaging, the control circuit 101moves onto step S208. At step S208, the control circuit 101 determineswhether to reset the in-focus positions. When the in-focus positionsneed resetting, the control circuit 101 returns to step S202. When thein-focus positions need no resetting, the control circuit 101 returns tostep S204.

For example, when the digital camera 100 performs imaging in a trackingmode in which the in-focus position is changed in real time according tothe position of a subject, the in-focus position also needs to bechanged in real time. In such a case, when there is no instruction forstarting imaging at step S207, the control circuit 101 returns to stepS202 to set the in-focus positions once more. Meanwhile, in the case ofsetting first in-focus positions at predetermined distances from thedigital camera 100 at step S202 and not changing the settings afterthat, there is no need to reset the in-focus positions. Therefore, insuch a case, when there is no instruction for starting imaging at stepS207, the control circuit 101 returns to step S204, not step S202.

At step S209, the imaging element 104 performs real imaging at theplurality of in-focus positions set at step S202.

At step S210, the image processing circuit 107 composites the imagescaptured by the real imaging at step S209. An example of the method forcompositing the images is as briefly described below. The imageprocessing circuit 107 creates a composition MAP using contrast valuesobtained from the images. Specifically, out of the plurality of images,the composition ratio of the image with the highest contrast value isset to 100%, and the composition ratio of the other images is set to 0%,in respective noticed areas or pixels. When the composition ratiochanges from 0% to 100% (or from 100% to 0%) between the adjacentpixels, the composition boundary becomes unnaturally prominent.Accordingly, a low-pass filter with a predetermined number of pixels(tap numbers) is applied to the composition MAP to process thecomposition MAP such that the composition ratio does not sharply changebetween the adjacent pixels. Alternatively, the composition MAP may becreated based on the contrast values of the images in the noticed areaor pixel such that the images with higher contrast values have highercomposition ratios.

In this way, according to the first embodiment, it is possible tocorrect live view display during imaging with consideration given to thefield angle of a composite image, and display the field viewcorresponding to the field angle of the composite image to the userduring imaging.

In the description of step S202, a predetermined number of in-focuspositions is set with reference to the in-focus position in the focusarea set by the user through the touch panel, but one or moreembodiments of the present disclosure are not limited to this.Alternatively, the area occupied by the subject with the referencein-focus position may be identified from the image and a plurality ofin-focus positions may be set based on range information in this area.

FIGS. 5A and 5B are diagrams for describing an example of an array ofsensors included in the imaging element 104 capable of acquiring rangeinformation on the subject. FIG. 5A illustrates a structure in whicheach pixel 500 has two photoelectric conversion units 501 and 502capable of independently reading optical signals. The number ofphotoelectric conversion units included in each pixel is not limited totwo, but may be three or more. For example, FIG. 5B illustrates astructure in which each pixel 510 has four photoelectric conversionunits 511 to 514. The following description is based on the structure inwhich one pixel has two photoelectric conversion units.

FIG. 6 is a diagram for describing the state of incidence of opticalsignals on a pixel having a plurality of photoelectric conversion units.

Referring to FIG. 6, a pixel array 601 includes micro lenses 602, colorfilters 603, and photoelectric conversion units 604 and 605. Thephotoelectric conversion units 604 and 605 belong to the same pixel andcorrespond to the common micro lens 602 and the common color filter 603.FIG. 6 is a top view of the digital camera 100 in which the twophotoelectric conversion units 604 and 605 corresponding to one pixelare aligned abreast. Out of light beams emitted from an exit pupil 606,with an optical axis 609 as a boundary, the upper light beam (equivalentto light beam from an area 607) enters the photoelectric conversion unit605, and the lower light beam (equivalent to light beam from an area608) enters the photoelectric conversion unit 604. That is, thephotoelectric conversion units 604 and 605 receive light from differentareas of the exit pupil in the imaging lens. When the signal received bythe photoelectric conversion unit 604 is designated as image A and thesignal received by the photoelectric conversion unit 605 as image B, itis possible to calculate a defocus amount based on the phase differencebetween the image A and the image B, and acquire range information. Inparticular, when pixels having two photoelectric conversion units aredisposed on the entire imaging element 104, the imaging element 104 canacquire range information at an arbitrary place of the subject.

FIG. 7 is a diagram for describing a method for acquiring rangeinformation on a subject by using pixels having a plurality ofphotoelectric conversion units. FIG. 7A illustrates 25 pixels in totalin a 5×5 array on a screen for visibility, but actual pixels are smallerin size and larger in number. The pixels superimposed on the subjectsuch as a pixel 701 are depicted by solid lines, and the other pixels bydotted lines like a pixel 702. The range information on the entiresubject can be acquired by acquiring range information on all thesolid-line pixels as described above.

Hereinafter, setting of the in-focus positions at step S202 in the firstembodiment will be described. The user performs a touch operation usingthe touch panel included in the operation member 110 to specify an area703 as a portion of the subject. When the area 703 is selected by thistouch operation, the control circuit 101 detects an entire area 704 ofthe subject to which the area 703 belongs. For example, the controlcircuit 101 analyzes color information in the image through the imageprocessing circuit 107 to detect the subject from the color information.Alternatively, the control circuit 101 may acquire edge information andrange information in the image through the image processing circuit 107and detect the entire area 704 of the subject based on the information.

Subsequently, as described above, the control circuit 101 acquires rangeinformation on the pixels superimposed on the entire area 704 of thesubject such as the pixel 701 illustrated in FIG. 7A. The controlcircuit 101 sets an in-focus position at the longest distance from thesubject as well as an in-focus position at the shortest distance fromthe subject, in the acquired range information. Then, the controlcircuit 101 sets some in-focus positions between the two in-focuspositions by a predetermined method such as equal spacing.

Second Embodiment

In a second embodiment, unlike the first embodiment, the control circuit101 does not automatically set the in-focus positions but the usermanually moves the in-focus position for imaging. Hereinafter, thesecond embodiment will be described. The descriptions of the samecomponents as those of the first embodiment will not be repeated.

FIG. 8 is a flowchart for describing the second embodiment.

At step S801, as at step S802 of the first embodiment, the controlcircuit 101 acquires lens information.

At step S802, the control circuit 101 determines whether the imagemagnification has a monotonic decrease or a monotonic increase. FIG. 9Ais a diagram illustrating the state in which the image magnification inthe second embodiment has a monotonic decrease, and FIG. 9B is a diagramillustrating the state in which the image magnification in the secondembodiment has a monotonic increase. The control circuit 101 analyzes afunction indicating the relationship between an image magnification andan in-focus position based on the lens information acquired at stepS801. When the image magnification decreases constantly as the in-focusposition becomes more distant as shown by a curve 901 in FIG. 9A, thecontrol circuit 101 determines that the image magnification has amonotonic decrease. In contrast, when the image magnification increasesconstantly as the in-focus position becomes more distant as shown by acurve 911 in FIG. 9B, the control circuit 101 determines that the imagemagnification has a monotonic increase. When determining at step S802that the image magnification has the monotonic decrease, the controlcircuit 101 moves onto step S803. When determining at step S802 that theimage magnification has the monotonic increase, the control circuit 101moves onto S804.

At step S803, the control circuit 101 restricts the drive direction ofthe in-focus position to the infinite distance side. That is, after thestart of the real imaging at step S806 described later, the controlcircuit 101 restricts the movement of the in-focus position to thedirection from an in-focus position 902 to an in-focus position 903along arrow 904 in FIG. 9A. When the in-focus position moves onto theopposite direction of the arrow 904, the control circuit 101 issues awarning as described later.

At step S804, the control circuit 101 restricts the drive direction ofthe in-focus position to the closest distance side. That is, after thestart of the real imaging, the control circuit 101 restricts themovement of the in-focus position to the direction from an in-focusposition 913 to an in-focus position 912 along arrow 914 in FIG. 9B.When the in-focus position moves to the opposite direction of the arrow914, the control circuit 101 issues a warning as described later.

The user adjusts the in-focus position of the optical system 103 andmoves the position to the initial in-focus position. The user can setthe initial in-focus position to an arbitrary position while watchingthe blurring level of the subject in live view or through an electronicviewfinder. At step S805, the control circuit 101 determines whetherthere is an instruction for first real imaging from the user. When thereis the instruction for real imaging, the control circuit 101 moves ontostep S806. When there is no instruction, the control circuit 101 entersa standby state at step S805.

At step S806, the imaging element 104 performs real imaging at theinitial in-focus position, and the control circuit 101 acquires theimage magnification corresponding to the in-focus position. The imagemagnification acquired by the control circuit 101 is equivalent to “themaximum set image magnification” in the first embodiment.

At step S807, the control circuit 101 determines whether the user ismoving the in-focus position in the direction restricted at step S803 orS804. For example, in the case of determining at step S802 that theimage magnification has a monotonic decrease and restricting at stepS803 the drive direction of the in-focus position to the infinitedistance side, when the in-focus position is being moved to the closestdistance side, the display 108 displays a warning at step S807.Alternatively, the display 108 displays a live view image at step S808and displays a warning on the displayed live view image. Stillalternatively, the digital camera 100 may issue a warning such as a beepor the like. In contrast, when the in-focus position is being moved tothe infinite distance side at step S807, the control circuit 101 movesonto step S809.

At step S809, the image processing circuit 107 corrects the live viewimage. By restricting the moving direction of the in-focus position atstep S807, the field angle of the live view image is greater than thefield angle of the image captured by the real imaging at the initialin-focus position. Accordingly, as at step S205 in the first embodiment,the image processing circuit 107 performs enlargement processing to cutand discard the portion not included in the displayed composite image.The specific method will not be described here because it is the same asthat at step S205.

At step S810, the display 108 displays the live view image after thecorrection by the image processing circuit 107 at step S809.

At step S811, the control circuit 101 determines whether there is aninstruction for real imaging from the user with the operation member110. For example, the control circuit 101 determines whether the userpressed a button in the operation member 110. When there is theinstruction for real imaging, the control circuit 101 moves onto stepS812 where the imaging element 104 performs imaging. When there is noinstruction for real imaging, the control circuit 101 returns to stepS807.

After the real imaging by the imaging element 104 at step S812, thecontrol circuit 101 determines at step S813 whether there is aninstruction for terminating the imaging. In this step, for example, thecontrol circuit 101 determines whether the user has pressed any buttonother than the button pressed at step S811. Alternatively, the controlcircuit 101 determines whether the user has provided the instruction forterminating the imaging through the touch panel. Still alternatively,the user may have determined the number of images to be captured so thatthe control circuit 101 determines at step S813 whether images equal innumber to the predetermined number of images to be captured have beencaptured by the real imaging. When the predetermined number of images tobe captured has been reached, the control circuit 101 determines thatthe imaging is to be terminated. When it is determined at Step S813 thatthe imaging is to be terminated, the control circuit 101 moves onto stepS814. When it is determined that the imaging is not to be terminated,the control circuit 101 returns to step S807.

At step 814, the image processing circuit 107 composites a plurality ofimages captured by the real imaging by the imaging element 104 at stepsS806 and S812. The method for compositing the images is the same as thatat step S209.

The foregoing description is merely for an example of the secondembodiment and the second embodiment can be modified in various manners.For example, at step S808, the control circuit 101 may issue a warningnot only when the in-focus position is being moved to a direction otherthan the restricted direction but also when the in-focus position hasmoved outside the depth of the subject at an adjacent in-focus position.Alternatively, when there are pixels 500 as illustrated in FIG. 5 of thesecond embodiment, the control circuit 101 may perform subject detectionas in the second embodiment, and when the in-focus position fallsoutside the range of subject distance in the optical axis direction, thecontrol circuit 101 may issue a warning.

According to the second embodiment, when the user manually performs realimaging at a plurality of in-focus positions, the field angle equivalentto the composite image can be displayed for the user at the initialin-focus position by restricting the movement direction of the in-focusposition.

Other Embodiments

The foregoing embodiments are carried out by a digital camera but one ormore embodiments of the present disclosure are not limited to a digitalcamera. For example, one or more embodiments of the present disclosuremay be carried out by a mobile device with an internal imaging elementor a network camera capable of image capturing.

One or more embodiments of the present disclosure may be carried out byproviding programs for executing one or more functions in the foregoingembodiments to a system or an apparatus via a network or a storagemedium, and reading and executing the programs by one or more processorsin the system or the apparatus. Alternatively, one or more embodimentsof the present disclosure may be carried out by a circuit (for example,ASIC) performing the one or more functions.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed embodiments. The scope of the following claimsis to be accorded the broadest interpretation so as to encompass allsuch modifications and equivalent structures and functions.

What is claimed is:
 1. An image processing method comprising: displaying a plurality of images obtained by picking up in different in-focus positions, the plurality of images having the different in-focus positions being used for generating a composite image; and performing a control of displaying of the obtained images corresponding to the different in-focus positions before generating the composite image such that each display indicates a portion corresponding to a field angle of the composite image.
 2. The image processing method according to claim 1, further comprising: generating the composite image using the plurality of images having the different in-focus positions.
 3. The image processing method according to claim 2, wherein in the generating, the composite image is generated by compositing the plurality of images in respective notice areas with the highest contrast among the plurality of images.
 4. The image processing method according to claim 3, wherein a depth field of the composite image is deeper than the depth field of any of the plurality of images.
 5. The image processing method according to claim 1, wherein a field angle of at least one of the plurality of images is different from a field angle of the composite image.
 6. The image processing method according to claim 1, wherein field angles of the plurality of images are different according to the in-focus positions of the plurality of images.
 7. The image processing method according to claim 6, wherein the field angle of the composite image is equal to a narrowest field angle among the field angles of the plurality of images.
 8. The image processing method according to claim 6, further comprising: acquiring a reference field angle out of the field angles of the plurality of images as the field angle corresponding to the field angle of the composite image, wherein in the performing the control of displaying, the each display is performed in accordance with the reference angle.
 9. The image processing method according to claim 8, wherein in the acquiring the reference field angle, optical system information relating to an optical system used for picking up the plurality of images is acquired and the reference field angle in accordance with the optical system information and the in-focus position is acquired.
 10. The image processing method according to claim 1, wherein in the performing the control of displaying, the portion corresponding to the field angle of the composite image is cut off from the obtained image and the cut-off portion is displayed.
 11. A non-transitory computer-readable medium for causing a computer to execute an image processing method, the image processing method comprising: displaying a plurality of images obtained by picking up in different in-focus positions, the plurality of images having the different in-focus positions being used for generating a composite image; and performing a control of displaying of the obtained images corresponding to the different in-focus positions before generating the composite image such that each display indicates a portion corresponding to a field angle of the composite image. 